What is claimed is: 



1 1 . A method for robustly producing a motion compensated interpolation video frame, the 

2 method comprising: 

3 constructing N motion compensated interpolated frames between two existing 

4 frames Fl and F2, wherein N comprises an integer with a value of at least 

5 2; and 

6 fusing the N motion compensated interpolated frames into a single motion 

7 compensated interpolated frame. 

1 2. The method of claim 1 wherein constructing each of the N motion compensated 

2 interpolated frames between the two existing frames Fl and F2 further comprises: 

3 selecting a first set of pixels in Fl, wherein the pixel sets selected for each of the 

4 N motion compensated frames varies; 

5 selecting a corresponding second set of pixels in F2, wherein the corresponding 

6 pixel sets selected for each of the N motion compensated frames varies; 

7 generating a first mesh for the first set of pixels and a second mesh for the second 

8 set of pixels; 

9 estimating a first flow of motion from the first set of pixels to the second set of 

10 pixels, and a second flow of motion from the second set of pixels to the 

11 first set of pixels; 

12 generating a first motion compensated mesh based on the first mesh and the first 

13 estimated flow of motion, and a second motion compensated mesh based 

14 on the second mesh and the second estimated flow of motion; 
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15 computing a first warped image by warping Fl using the first mesh and the first 

16 motion compensated mesh, and a second warped image by warping F2 

17 using the second mesh and the second motion compensated mesh; and 

1 8 linearly combing the first warped image and the second warped image. 

1 3. The method of claim 2 wherein selecting a set of pixels in a frame further comprises: 

2 classifying some pixels in the frame as having high spatial frequency contents and 

3 selecting the classified pixels, wherein the specific classification criteria is 

4 different for each of the N motion compensated frames. 

1 4. The method of claim 2 wherein generating a mesh for a set of pixels further 

2 comprises: 

3 fitting a polygonal mesh to the set of pixels. 

1 5. The method of claim 4 wherein fitting a polygonal mesh to a set of pixels further 

2 comprises: 

3 applying a Delaunay tri angulation to the set of pixels, using edges of the 

4 associated frame as imposed boundaries. 

1 6. The method of claim 2 wherein estimating a flow of motion between two sets of pixels 

2 further comprises : 

3 applying an optical flow constraint equation to the first set of pixels, the optical 

4 flow constraint equation comprising x*u + y*v + t = 0, wherein u and v 

5 are unknown components of the flow and x, y and t stand for 

6 differentiation. 
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1 7. The method of claim 1 wherein fusing the N motion compensated interpolated frames 

2 into a single, motion compensated interpolated frame further comprises: 

3 for each pixel x, y in a final fused motion compensated interpolated frame: 

4 applying a scalar median filter componentwise to a corresponding pixel in 

5 each of the N motion compensated interpolated frames 1-N to 

6 produce a resulting pixel x', y f ; and 

7 setting x, y to a corresponding pixel from a one of the N motion 

8 compensated interpolated frames that is color nearest to x f , y\ 

1 8. A computer readable medium containing a computer program product for robustly 

2 producing a motion compensated interpolation video frame, the computer program product 

3 comprising: 

4 program code for constructing N motion compensated interpolated frames 

5 between two existing frames Fl and F2, wherein N comprises an integer 

6 with a value of at least 2; and 

7 program code for fusing the N motion compensated interpolated frames into a 

8 single motion compensated interpolated frame. 

1 9. The computer readable medium of claim 8 wherein the program code for constructing 

2 each of the N motion compensated interpolated frames between the two existing frames Fl and 

3 F2 further comprises: 

4 program code for selecting a first set of pixels in Fl , wherein the pixel sets 

5 selected for each of the N motion compensated frames varies; 



11 



6 program code for selecting a corresponding second set of pixels in F2, wherein 

7 the corresponding pixel sets selected for each of the N motion 

8 compensated frames varies; 

9 program code for generating a first mesh for the first set of pixels and a second 

1 0 mesh for the second set of pixels; 

1 1 program code for estimating a first flow of motion from the first set of pixels to 

12 the second set of pixels, and a second flow of motion from the second set 

1 3 of pixels to the first set of pixels; 

14 program code for generating a first motion compensated mesh based on the first 

15 mesh and the first estimated flow of motion, and a second motion 

16 compensated mesh based on the second mesh and the second estimated 

17 flow of motion; 

1 8 program code for computing a first warped image by warping Fl using the first 

19 mesh and the first motion compensated mesh, and a second warped image 

20 by warping F2 using the second mesh and the second motion compensated 

21 mesh; and 

22 program code for linearly combing the first warped image and the second warped 

23 image. 

1 10. The computer readable medium of claim 9 wherein the program code for selecting a 

2 set of pixels in a frame further comprises: 

3 program code for classifying some pixels in the frame as having high spatial 

4 frequency contents and selecting the classified pixels, wherein the specific 
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5 classification criteria is different for each of the N motion compensated 

6 frames. 

1 11. The computer readable medium of claim 9 wherein the program code for generating 

2 a mesh for a set of pixels further comprises: 

3 program code for fitting a polygonal mesh to the set of pixels. 

1 12. The computer readable medium of claim 1 1 wherein the program code for fitting a 

2 polygonal mesh to a set of pixels further comprises: 

3 program code for applying a Delaunay triangulation to the set of pixels, using 

4 edges of the associated frame as imposed boundaries. 

1 13. The computer readable medium of claim 9 wherein the program code for estimating 

2 a flow of motion between two sets of pixels further comprises: 

3 program code for applying an optical flow constraint equation to the first set of 

4 pixels, the optical flow constraint equation comprising x*u + y*v + t = 

5 0, wherein u and v are unknown components of the flow and x, y and t 

6 stand for differentiation. 

1 14. The computer readable medium of claim 8 wherein the program code for fusing the 

2 N motion compensated interpolated frames into a single, motion compensated interpolated frame 

3 further comprises: 

4 program code for, for each pixel x, y in a final fused motion compensated 

5 interpolated frame: 
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6 applying a scalar median filter componentwise to a corresponding pixel in 

7 each of the N motion compensated interpolated frames 1-N to 

8 produce a resulting pixel x\ /; and 

9 setting x, y to a corresponding pixel from a one of the N motion 

10 compensated interpolated frames that is color nearest to x\ y\ 

1 1 5. A computer system for robustly producing a motion compensated interpolation video 

2 frame, the computer system comprising: 

3 means for constructing N motion compensated interpolated frames between two 

4 existing frames Fl and F2, wherein N comprises an integer with a value of 

5 at least 2; and 

6 means for fusing the N motion compensated interpolated frames into a single 

7 motion compensated interpolated frame. 

1 16. The computer system of claim 15 wherein the means for constructing each of the N 

2 motion compensated interpolated frames between the two existing frames Fl and F2 further 

3 comprises: 

4 means for selecting a first set of pixels in Fl, wherein the pixel sets selected for 

5 each of the N motion compensated frames varies; 

6 means for selecting a corresponding second set of pixels in F2, wherein the 

7 corresponding pixel sets selected for each of the N motion compensated 

8 frames varies; 

9 means for generating a first mesh for the first set of pixels and a second mesh for 
10 the second set of pixels; 
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1 1 means for estimating a first flow of motion from the first set of pixels to the 

12 second set of pixels, and a second flow of motion from the second set of 

1 3 pixels to the first set of pixels; 

14 means for generating a first motion compensated mesh based on the first mesh 

15 and the first estimated flow of motion, and a second motion compensated 

16 mesh based on the second mesh and the second estimated flow of motion; 

17 means for computing a first warped image by warping Fl using the first mesh and 

1 8 the first motion compensated mesh, and a second warped image by 

19 warping F2 using the second mesh and the second motion compensated 

20 mesh; and 

21 means for linearly combing the first warped image and the second warped image. 

1 17. The computer system of claim 16 wherein the means for selecting a set of pixels in a 

2 frame further comprises: 

3 means for classifying some pixels in the frame as having high spatial frequency 

4 contents and selecting the classified pixels, wherein the specific 

5 classification criteria is different for each of the N motion compensated 

6 frames. 

1 18. The computer system of claim 16 wherein the means for generating a mesh for a set 

2 of pixels further comprises: 

3 means for fitting a polygonal mesh to the set of pixels. 

1 19. The computer system of claim 1 8 wherein the means for fitting a polygonal mesh to 

2 a set of pixels further comprises: 
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3 means for applying a Delaunay triangulation to the set of pixels, using edges of 

4 the associated frame as imposed boundaries. 

1 20. The computer system of claim 16 wherein the means for estimating a flow of motion 

2 between two sets of pixels further comprises: 

3 means for applying an optical flow constraint equation to the first set of pixels, 

4 the optical flow constraint equation comprising x*u + y*v + t = 0, 

5 wherein u and v are unknown components of the flow and x, y and t stand 

6 for differentiation. 

1 21 . The computer system of claim 1 5 wherein the means for fusing the N motion 

2 compensated interpolated frames into a single, motion compensated interpolated frame further 

3 • comprises: 

4 means for, for each pixel x, y in a final fused motion compensated interpolated 

5 frame: 

6 applying a scalar median filter componentwise to a corresponding pixel in 

7 each of the N motion compensated interpolated frames 1-N to 

8 produce a resulting pixel x', y'; and 

9 setting x, y to a corresponding pixel from a one of the N motion 

10 compensated interpolated frames that is color nearest to x ! , y\ 

1 22. A computer system for robustly producing a motion compensated interpolation video 

2 frame, the computer system comprising: 
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3 a software portion configured to construct N motion compensated interpolated 

4 frames between two existing frames Fl and F2, wherein N comprises an 

5 integer with a value of at least 2; and 

6 a software portion configured to fuse the N motion compensated interpolated 

7 frames into a single motion compensated interpolated frame. 

1 23. The computer system of claim 22 wherein the software portion configured to 

2 construct each of the N motion compensated interpolated frames between the two existing 

3 frames Fl and F2 further comprises: 

4 a software portion configured to select a first set of pixels in Fl , wherein the pixel 

5 sets selected for each of the N motion compensated frames varies; 

6 a software portion configured to select a corresponding second set of pixels in F2, 

7 wherein the corresponding pixel sets selected for each of the N motion 

8 compensated frames varies; 

9 a software portion configured to generate a first mesh for the first set of pixels and 

10 a second mesh for the second set of pixels; 

11 a software portion configured to estimate a first flow of motion from the first set 

12 of pixels to the second set of pixels, and a second flow of motion from the 

13 second set of pixels to the first set of pixels; 

14 a software portion configured to generate a first motion compensated mesh based 

1 5 on the first mesh and the first estimated flow of motion, and a second 

16 motion compensated mesh based on the second mesh and the second 

1 7 estimated flow of motion; 
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a software portion configured to compute a first warped image by warping Fl 

using the first mesh and the first motion compensated mesh, and a second 
warped image by warping F2 using the second mesh and the second 
motion compensated mesh; and 

a software portion configured to linearly combine the first warped image and the 
second warped image. 



1 24. The computer system of claim 23 wherein the software portion configured to select a 

2 set of pixels in a frame further comprises: 

3 a software portion configured to classify some pixels in the frame as having high 

4 spatial frequency contents and selecting the classified pixels, wherein the 

5 specific classification criteria is different for each of the N motion 

6 compensated frames. 

1 25. The computer system of claim 23 wherein the software portion configured to 

2 generate a mesh for a set of pixels further comprises: 

3 a software portion configured to fit a polygonal mesh to the set of pixels. 

1 26. The computer system of claim 25 wherein the software portion configured to fit a 

2 polygonal mesh to a set of pixels further comprises: 

3 a software portion configured to apply a Delaunay triangulation to the set of 

4 pixels, using edges of the associated frame as imposed boundaries. 

1 27. The computer system of claim 23 wherein the software portion configured to 

2 estimate a flow of motion between two sets of pixels further comprises: 
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3 a software portion configured to apply an optical flow constraint equation to the 

4 first set of pixels, the optical flow constraint equation comprising x * u + y 

5 * v + 1 = 0, wherein u and v are unknown components of the flow and x, y 

6 and t stand for differentiation. 

1 28. The computer system of claim 22 wherein the software portion configured to fuse the 

2 N motion compensated interpolated frames into a single, motion compensated interpolated frame 

3 further comprises: 

4 a software portion configured to, for each pixel x, y in a final fused motion 

5 compensated interpolated frame: 

6 apply a scalar median filter componentwise to a corresponding pixel in 

7 each of the N motion compensated interpolated frames 1-N to 

8 produce a resulting pixel x\ y f ; and 

9 set x, y to a corresponding pixel from a one of the N motion compensated 
10 interpolated frames that is color nearest to x\ y\ 
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